In order to improve performance or reliability, a redundant structure including a plurality of control modules for controlling inputting data to and outputting data from a storage device which is a storage medium such as a hard disk drive is adopted in conventional storage systems including distributed cache memory type storage apparatus. Each control module performs the process of reading data from and writing data to a logical volume.
In preparation for the occurrence of an unexpected disaster or the like, some of these storage apparatus have a REC (Remote Equivalent Copy) function for transferring data in a logical volume to a remote place via a network and performing remote copy. With an ordinary REC function, however, order is not guaranteed at the time of transferring data. Therefore, it may take a lot of time to recover the data after the occurrence of a disaster. In order to solve this problem, some storage systems have, for example, a REC consistency mode as a remote copy function which guarantees the order of remote copy.
In the REC consistency mode, a storage apparatus stores data sent from a host computer in a buffer of a control module which is a dedicated memory referred to as a consistency buffer and which is divided into constant-size storage areas according to generation. The storage apparatus then transfers the data to a storage apparatus which is a copy destination via a network according to generation. The storage apparatus to which the data is transferred stores the data in a logical volume so that the order of the data will be maintained. By doing so, the order of the remote copy is guaranteed.
FIGS. 30 and 31 illustrate a storage system having a remote copy function which guarantees order. In a storage system illustrated in FIG. 30, a storage apparatus 3100 including control modules #00 and #01 and a storage apparatus 3200 including control modules #10 and #11 are connected via a network 3011. It is assumed that remote copy is performed via the network 3011 with the storage apparatus 3100 and 3200 as a copy source and a copy destination respectively.
The storage apparatus 3100 has the control module #00 including a record only buffer 31180 including a buffer 31180a associated with a storage device 31170 and a BIT (Buffer Index Table) storage section 31180b, a buffer set information storage section 31190 which stores information regarding a buffer set, and the storage device 31170 which stores data and the control module #01 including a record only buffer 31181 including a buffer 31181a associated with a storage device 31171 and a BIT storage section 31181b, a buffer set information storage section 31191, and the storage device 31171.
Similarly, the storage apparatus 3200 has the control module #10 including a record only buffer 32180 including a buffer 32180a associated with a storage device 32170 and a BIT storage section 32180b, a buffer set information storage section 32190, and the storage device 32170 and the control module #11 including a record only buffer 32181 including a buffer 32181a associated with a storage device 32171 and a BIT storage section 32181b, a buffer set information storage section 32191, and the storage device 32171.
Each of the buffers 31180a, 31181a, 32180a, and 32181a is divided into a plurality of constant-size storage areas. A buffer ID is assigned to each of the storage areas obtained by dividing the buffers 31180a, 31181a, 32180a, and 32181a. Data which has already been stored in the corresponding storage devices 31170, 31171, 32170, and 32171 or data which is to be stored in the corresponding storage devices 31170, 31171, 32170, and 32171 is stored temporarily in the storage areas of the buffers 31180a, 31181a, 32180a, and 32181a. In addition, the BIT storage sections 31180b, 31181b, 32180b, and 32181b store BITs which are information including storage locations on the storage devices 31170, 31171, 32170, and 32171 where the data stored in the corresponding buffers 31180a, 31181a, 32180a, and 32181a is stored, buffer IDs, and the like.
In FIG. 30, “(0000)” indicated in the buffer 31180a represents buffer data stored in a storage area the buffer ID of which is “0000.” “0000” indicated in the BIT represents a buffer ID of a storage area corresponding to the BIT.
Each of the buffer set information storage sections 31190, 31191, 32190, and 32191 stores information regarding a buffer set which is a combination of buffer data stored in storage areas of buffers included in control modules. For example, information regarding a buffer set which is a combination of the buffer data (0000) and the buffer data (0100) enclosed with a dotted line in the horizontal direction in the storage apparatus 3100 (copy source) is stored.
Information regarding a buffer set includes information for associating buffer data stored in a storage area of a buffer of a control module corresponding to a buffer set information storage section with a storage area of a buffer of the storage apparatus 3200 (copy destination) which stores the buffer data. For example, the buffer data (0000) stored in the storage area the buffer ID of which is “0000” is associated with a storage area the buffer ID of which is “1000” and to which the buffer data (0000) is copied in the buffer set information storage section 31190 of the control module #00.
When the control modules #00 and #01 of the storage apparatus 3100 in the above storage system accept a write I/O instruction (instruction to write received data to a storage device) from, for example, a host computer, the control modules #00 and #01 store the data in the storage devices 31170 and 31171, respectively, in accordance with the write I/O instruction. In addition, the control modules #00 and #01 store the data in the buffers. At this time the data is managed by the buffer set. That is to say, pieces of buffer data are associated with each other by buffer set information and make up a buffer set. When the control modules #00 and #01 complete writing the data to the buffers, the control modules #00 and #01 begin to transfer the data to the storage apparatus 3200 by the buffer set, that is to say, to perform remote copy.
The data transferred from the storage apparatus 3100 (copy source) to the storage apparatus 3200 by the remote copy is stored in the buffers included in the control modules #10 and #11 of the storage apparatus 3200 (copy destination) in accordance with the buffer set information. The control modules #10 and #11 then write the data stored in the buffers to the storage devices 32170 and 32171 respectively. When this process is completed, each control module included in the storage apparatus 3100 and 3200 frees a buffer and the like.
As has been described, data is sent in block by the use of the record only buffers and a buffer set is controlled in block. In the storage apparatus 3200 (copy destination) the data is written to the storage devices 32170 and 32171 by the buffer set. By doing so, the order of the data transferred is guaranteed.    Japanese Laid-open Patent Publication No. 2001-100939    Japanese Laid-open Patent Publication No. 07-121308    Japanese Laid-open Patent Publication No. 05-100927    Japanese Laid-open Patent Publication No. 2006-260292    International Publication Pamphlet No. WO/2008/152691
As illustrated in FIG. 31, however, the buffers and the BITs in the storage apparatus 3100 (copy source) which store data that has been transferred wait for freeing until the data is expanded in the storage devices of the storage apparatus 3200 (copy destination). The data is held in the storage apparatus 3100 (copy source).
It is assumed that there is a delay in the process of expanding the data in the storage devices because of a heavy update load on a logical volume of the storage apparatus 3200 or that an abnormality has occurred in the network 3011 between the storage apparatus 3100 and 3200. For example, if the line capability of the network 3011 is poor or is not uniform, then a delay occurs in a transfer process.
If a delay occurs in a transfer process from such a cause, time for which the buffers and the BITs in the storage apparatus 3100 and 3200 are used becomes longer and the buffers in the storage apparatus 3100 and 3200 cannot be freed.
It is assumed that the storage apparatus 3100 accepts new write I/O instructions from, for example, the host computer in this state. Each time the storage apparatus 3100 (copy source) accepts the write I/O instruction, the storage apparatus 3100 stores data to be written in the buffers. As a result, the buffers of the storage apparatus 3100 are used one after another.
If this state continues, it is impossible to ensure a buffer in the storage apparatus 3100 which can store data. As a result, the buffers are exhausted in the storage apparatus 3100. In addition, if data the size of which is larger than the set size of the buffers is processed, then the buffers are also exhausted in the storage apparatus 3100.
In a state in which the buffers are exhausted, the storage apparatus 3100 does not execute a write I/O instruction from, for example, the host computer and is in a stopped state. The storage apparatus 3100 cannot continue this state for a long time.
Accordingly, a buffer halt process is performed. That is to say, the execution of the write I/O instruction is stopped temporarily. If the state in which the buffers are exhausted is not canceled after the elapse of a certain period of time, then the execution is stopped temporarily and contents stored in the buffers are erased. The storage apparatus 3100 and 3200 perform the buffer halt process to clear the buffers. In addition, the storage apparatus 3100 and 3200 resume the execution of the write I/O instruction.
At this time the storage apparatus 3100 writes back information, such as data, in the buffers to a dedicated bit map. After the storage apparatus 3100 performs the buffer halt process, the storage apparatus 3100 performs remote copy transfer which does not guarantee order in accordance with the bit map. In this case, remote copy which guarantees order is stopped.
A storage system having a REC disk buffered consistency mode will now be discussed. With the above remote copy function which guarantees order, data transferred by remote copy is stored temporarily in a save buffer, such as a hard disk drive, in the REC disk buffered consistency mode. By utilizing this mode, order can be guaranteed at remote copy time, for example, even if an update load on a logical volume of a storage apparatus is heavy or an abnormality has occurred in transfer via a network between storage apparatus. It is assumed that, in this storage system, data is stored in a save buffer according to generation.
In this storage system, a buffer included in each control module is used for writeback and staging. The writeback is the operation of reading out data from a buffer and writing the data to a save buffer. The staging is the operation of reading out data from a save buffer and writing the data to a buffer. With the writeback and the staging, access to a save buffer is performed in prescribed order in order to guarantee the order of data. For example, with the writeback and the staging, access to a save buffer is performed in order of generation. However, if access to a save buffer is performed according to generation in accordance with the order, areas of the save buffer which are not consecutive may be accessed by the writeback. This may make it impossible to realize the original performance of the save buffer. As a result, a delay in the writeback leads to a delay in storing data in buffers and the order may not be guaranteed adequately.